Papers with word embeddings
Probabilistic FastText for Multi-Sense Word Embeddings (P18-1)
Copied to clipboard
| Challenge: | Probabilistic FastText model for word embeddings captures word senses, sub-word structure, and uncertainty information. |
| Approach: | They propose a model for word embeddings that captures multiple word senses . they represent each word with a Gaussian mixture density, with each vector representing an n-gram . |
| Outcome: | The proposed model outperforms dictionary-level probabilistic embeddings on word-similarity benchmarks. |
Enhanced Word Representations for Bridging Anaphora Resolution (N18-2)
Copied to clipboard
| Challenge: | Existing word representations do not capture semantic similarity for bridging anaphora resolution. |
| Approach: | They propose to use word embeddings to capture semantic similarity by exploring syntactic structure of noun phrases. |
| Outcome: | The proposed model achieves 30% of accuracy for bridging anaphora resolution on ISNotes corpus. |
Context-Aware Neural Machine Translation Decoding (D19-65)
Copied to clipboard
| Challenge: | Existing approaches to enhance neural machine translation systems to take into account document-level information make the training process slower or require document- level annotated data. |
| Approach: | They propose a decoding architecture that fuses the semantic space language model and a neural translation model. |
| Outcome: | The proposed approach improves translation quality for English–Spanish using BLEU and METEOR. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Bias and Fairness in Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | a tutorial will review the history of bias and fairness studies in machine learning and language processing . |
| Approach: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it presents recent community effort to quantify and mitigat bias in natural language processing models . |
| Outcome: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it aims to quantify and mitigate bias in natural language processing models for a wide spectrum of tasks . |
Opinion Mining with Deep Contextualized Embeddings (N19-3)
Copied to clipboard
| Challenge: | Existing methods for opinion expression detection are based on token-level sequence labeling . |
| Approach: | They propose to use BERT and conditional random field embedders to detect opinion expressions. |
| Outcome: | The proposed model outperforms ELMo embedders in opinion expression detection. |
A Bag-of-concepts Model Improves Relation Extraction in a Narrow Knowledge Domain with Limited Data (N19-3)
Copied to clipboard
| Challenge: | Existing methods for relation extraction on small data sets are time-consuming and expensive. |
| Approach: | They propose an automatic relation extraction task with limited annotated data and a narrow knowledge domain. |
| Outcome: | The proposed method outperforms methods of higher complexity on a small clinical corpus. |
Learning Word Embeddings for Data Sparse and Sentiment Rich Data Sets (N18-4)
Copied to clipboard
| Challenge: | Existing word embeddings for sentiment analysis are limited in domain specific applications . generic word embeds are poor initialization for tasks on domain specific data sets. |
| Approach: | They propose to use word embeddings adapted for domain specific data sets in sentiment classification applications. |
| Outcome: | The proposed algorithms learn word embeddings on sparse and sentiment rich data sets. |
Correlations between Word Vector Sets (D19-1)
Copied to clipboard
| Challenge: | Similarity measures based on word embeddings are easily competing with deep learning and expert-engineered systems on unsupervised semantic textual similarity tasks. |
| Approach: | They propose a new approach to measure word embeddings using pooling operations and correlation coefficients instead of pooling . they also propose centered kernel alignment as a natural generalisation of squared cosine similarity for sets of word vectors. |
| Outcome: | The proposed approach outperforms most recent methods while being much faster and trivial to implement. |
Generating Text through Adversarial Training Using Skip-Thought Vectors (N19-3)
Copied to clipboard
| Challenge: | Existing approaches to use word embeddings for text generation have been limited. |
| Approach: | They propose to use GANs with word embeddings to reproduce writing style in text . they use a sentence embeddable vector to model people's way of expression . |
| Outcome: | The proposed model outperforms baseline text generation networks across several metrics including BLEU-n, METEOR and ROUGE. |
Sentiment as an Ordinal Latent Variable (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing dictionaries are limited in coverage and sentiment scales vary widely; some are discrete others continuous. |
| Approach: | They propose a Bayesian generative model that learns a composite sentiment dictionary as an interpolation between six existing dictionaries with different scales. |
| Outcome: | The proposed model learns a composite sentiment dictionary as an interpolation between six existing dictionaries with different scales. |
Investigating the Frequency Distortion of Word Embeddings and Its Impact on Bias Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has shown that static word embeddings can encode words’ frequencies, but little has been studied about this behavior. |
| Approach: | They propose to use static word embeddings to encode words' frequencies and to assess the impact of this relationship on embeddable bias metrics. |
| Outcome: | The proposed model shows that word embeddings can produce higher similarity between high-frequency words than other embeddables. |
When Polysemy Matters: Modeling Semantic Categorization with Word Embeddings (2022.starsem-1)
Copied to clipboard
| Challenge: | Recent work using word embeddings to model semantic categorization has shown that static models outperform contextual models. |
| Approach: | They consider polysemy as a possible confounding factor in categorization decisions . they compare sense-level embeddings with previously studied static embedds . |
| Outcome: | The proposed model outperforms static models on coarse- and fine-grained categorization tasks. |
Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment (2022.findings-emnlp)
Copied to clipboard
Tuan Dinh, Jy-yong Sohn, Shashank Rajput, Timothy Ossowski, Yifei Ming, Junjie Hu, Dimitris Papailiopoulos, Kangwook Lee
| Challenge: | Recent studies show that unsupervised word translation is more accurate and robust without parallel corpora. |
| Approach: | They propose a method for unsupervised word translation that leverages visual observations and pretrained language-image models to align words. |
| Outcome: | The proposed method improves on the state-of-the-art language-image pretraining method for bilingual word alignment. |
A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus (2024.acl-long)
Copied to clipboard
| Challenge: | Natural language inference (NLI) is an actively studied topic serving as a proxy for natural language understanding. |
| Approach: | They propose to use a Romanian NLI corpus to analyze sentence pairs . they use multiple machine learning methods to establish competitive baselines . |
| Outcome: | The proposed model improves on the best model by employing a new curriculum learning strategy based on data cartography. |
Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing research has focused on applying semantic models to decode brain activity associated with the meaning of individual words. |
| Approach: | They evaluate a range of semantic models to capture metaphor processing in the brain . they found that compositional models and word embeddings capture differences in the processing of literal and metaphoric sentences . |
| Outcome: | The proposed models capture differences in the processing of literal and metaphoric sentences, providing support for the idea that the literal meaning is not fully accessible during familiar metaphor comprehension. |
Discontinuous Combinatory Constituency Parsing (2023.tacl-1)
Copied to clipboard
| Challenge: | Discontinuous parsing is more challenging than continuous parsers because children can group with syntactic cousins in the sentence rather than its two adjacent neighbors. |
| Approach: | They extend a pair of combinator-based constituency parsers into a discontinuous pair . they use a swap action and biaffine attention to iteratively compose constituent vectors from word embeddings without any grammar constraints. |
| Outcome: | The proposed parsers achieve state-of-the-art discontinuous accuracy with a significant speed advantage over continuous parsing. |
BERT-Based Neural Collaborative Filtering and Fixed-Length Contiguous Tokens Explanation (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing models that learn accurate representations of users and items are based on ratings, which oversimplify user preferences and item characteristics. |
| Approach: | They propose a novel, accurate, and explainable recommender model that integrates three key elements: BERT, multilayer perceptron, and maximum subarray problem to derive contextualized review features, model user-item interactions, and generate explanations. |
| Outcome: | The proposed model outperforms state-of-the-art models by an improvement gain of nearly 7% based on the human judges’ assessment . |
From Text to Lexicon: Bridging the Gap between Word Embeddings and Lexical Resources (C18-1)
Copied to clipboard
| Challenge: | Distributional word representations are omnipresent in modern NLP. |
| Approach: | They propose to combine lemmatization and part of speech (POS) typing to improve word embedding performance. |
| Outcome: | The proposed methods improve word embedding performance on verbs and verbs. |
LINSPECTOR WEB: A Multilingual Probing Suite for Word Representations (D19-3)
Copied to clipboard
| Challenge: | LINSPECTOR WEB is an open source multilingual inspector to analyze word embeddings. |
| Approach: | They propose to use LINSPECTOR WEB to analyze word embeddings in 28 languages. |
| Outcome: | The system performs 16 simple linguistic probing tasks for a diverse set of 28 languages. |
BOUN-ISIK Participation: An Unsupervised Approach for the Named Entity Normalization and Relation Extraction of Bacteria Biotopes (D19-57)
Copied to clipboard
| Challenge: | In 2011, the Bacteria Biotope Task was conducted for the first time as a part of the BioNLP Shared Task targeting the extraction of useful information regarding bacteria and their habitats. |
| Approach: | They propose two systems for the normalization of entities and the identification of relations between entities given a biomedical text. |
| Outcome: | The proposed method performs as good as deep learning based methods which require labeled data. |
Denoising Word Embeddings by Averaging in a Shared Space (2021.starsem-1)
Copied to clipboard
| Challenge: | Continuous word embeddings have been introduced several years ago as a standard building block for NLP tasks. |
| Approach: | They propose a method of fusing word embeddings that were trained on the same corpus but with different initializations. |
| Outcome: | The proposed method improves word embeddings on a range of tasks. |
Sudden Semantic Shifts in Swedish NATO discourse (2023.acl-srw)
Copied to clipboard
| Challenge: | Using word embeddings, we study sudden semantic shifts that occur when a sudden event radically changes public opinion on a topic. |
| Approach: | They use word embeddings to study how Twitter associations evolve . they find domain knowledge and data selection are of prime importance when using word embeds to understand semantic shifts. |
| Outcome: | The proposed method validates associations on Twitter with NATO in real-world events but is difficult to distinguish between noise and real-time signals. |
Topic Modeling in Embedding Spaces (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing topic models fail to learn interpretable topics when working with large and heavy-tailed vocabularies. |
| Approach: | They propose an embedded topic model that integrates word embeddings with a categorical distribution that is the natural parameter between the word’s embeddment and an embeddement of its assigned topic. |
| Outcome: | The embedded topic model outperforms existing topic models in terms of topic quality and predictive performance. |
A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings (2020.tacl-1)
Copied to clipboard
| Challenge: | Experimental results show that the proposed model outperforms word-level embedding methods in word similarity evaluation and word sense disambiguation. |
| Approach: | They propose a generative model that explores local and global context for joint learning topics and topic-specific word embeddings. |
| Outcome: | The proposed model outperforms word-level embedding methods in word similarity evaluation and word sense disambiguation. |
Joint Learning of Sense and Word Embeddings (L18-1)
Copied to clipboard
| Challenge: | Existing methods for learning lower-dimensional representations of words using unlabelled data learn a single representation for a word, ignoring the different senses of that word (polysemy). |
| Approach: | They propose a method that jointly learns sense-aware word embeddings using both unlabelled and sense-tagged text corpora. |
| Outcome: | The proposed method outperforms competing methods on word similarity and short-text classification benchmark datasets. |
Enhancing Word Embeddings with Knowledge Extracted from Lexical Resources (2020.acl-srw)
Copied to clipboard
| Challenge: | In this paper, we present an effective method for semantic specialization of word vector representations. |
| Approach: | They propose a method for semantic specialization of word vector representations using BabelNet. |
| Outcome: | The proposed method improves on word similarity and dialog state tracking tasks. |
Learning Visually Grounded Sentence Representations (N18-1)
Copied to clipboard
| Challenge: | Unsupervised sentence representation models suffer from the grounding problem because of lack of association between symbols and external information. |
| Approach: | They train a sentence encoder to predict image features of a caption and use them as sentence representations. |
| Outcome: | The proposed model improves on word embeddings and word representations on standard benchmarks. |
The Word Analogy Testing Caveat (N18-2)
Copied to clipboard
| Challenge: | a number of word analogy tests are used to evaluate word embeddings . word embeds are used as a proxy for semantics and syntax à la Harris . |
| Approach: | They propose to use word embeddings as a proxy for distributional similarity . they propose to apply a transfer learning approach to word embeds to improve performance . |
| Outcome: | The proposed method improves performance across a wide range of NLP tasks. |
ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems (N19-1)
Copied to clipboard
| Challenge: | Existing methods to regularize neural machine translation are limited in low-resource settings. |
| Approach: | They propose a method that uses regressing word embeddings to regularize neural machine translation. |
| Outcome: | The proposed system improves on a strong baseline and a state-of-the-art system. |
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (P18-1)
Copied to clipboard
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, Lawrence Carin
| Challenge: | Existing deep learning architectures to model compositionality in text sequences require a large number of parameters and expensive computations. |
| Approach: | They propose two additional pooling strategies over word embeddings for improved interpretability and hierarchical pooling for spatial (n-gram) information within text sequences. |
| Outcome: | The proposed pooling strategies improve interpretability and preserve spatial (n-gram) information within text sequences. |
Affective Retrofitted Word Embeddings (2022.aacl-main)
Copied to clipboard
| Challenge: | Word embeddings do not capture affective dimensions of valence, arousal, and dominance . valency, valance, and adolescence are present in words, but are not represented in text . |
| Approach: | They propose a method for updating word embeddings for affective meaning . they use a non-linear transformation function that maps pre-trained embedders to an affective vector space . |
| Outcome: | The proposed method improves inter-cluster and intra-c cluster distances for emotion-bearing words. |
attr2vec: Jointly Learning Word and Contextual Attribute Embeddings with Factorization Machines (N18-1)
Copied to clipboard
| Challenge: | popular word embeddings are used to learn vector representations from the context of words. |
| Approach: | They propose a framework for jointly learning embeddings for words and contextual attributes based on factorization machines. |
| Outcome: | The proposed framework improves on a text classification task compared to learning embeddings independently. |
On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach (2023.acl-short)
Copied to clipboard
| Challenge: | Word embeddings have been used to quantify biases in texts for years, but their statistical properties and advantages have not been studied. |
| Approach: | They propose to use PMI-based metric to quantify bias in corpora by conditional probabilities and odds ratio to approximate it. |
| Outcome: | The proposed measure can be approximated by an odds ratio, which makes statistical inferences cost-effective and meaningful. |
Revisiting Representation Degeneration Problem in Language Modeling (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Language modeling is a fundamental task in natural language processing, applications include machine translation, image captioning and speech recognition. |
| Approach: | They propose a cosine regularization method to solve the representation degeneration problem by analyzing the limitations of the proposed method and then propose an alternative regularization technique to tackle the problem. |
| Outcome: | The proposed method is effective in language modeling and image captioning. |
Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora (2020.acl-main)
Copied to clipboard
| Challenge: | comparing two corpus texts and searching for words that differ in their usage between them is a common problem in digital humanities and computational social science. |
| Approach: | They propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word. |
| Outcome: | The proposed method is interpretable and stable in 9 different setups and is highly reliable. |
Embedding derived animacy rankings offer insights into the sources of grammatical animacy (2025.naacl-long)
Copied to clipboard
| Challenge: | a generative linguistics perspective posits that grammar is shaped by innate cognitive biases. |
| Approach: | They applied the semantic projection approach to animacy, a feature that has not been previously explored using this method. |
| Outcome: | The proposed method is effective in deriving proxies of human perception from word embeddings and provides insights into the sources of grammatical animacy. |
Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing methods to debias word embeddings in binary settings such as gender and religion are limited to binary labels, whereas word2vec embedders can be used to propagate biases. |
| Approach: | They propose a method to debias word embeddings in multiclass settings such as gender and religion, extending the work of Bolukbasi et al. (2016). |
| Outcome: | The proposed method maintains the efficacy in standard NLP tasks while maintaining the utility of embeddings. |
An Analysis of Euclidean vs. Graph-Based Framing for Bilingual Lexicon Induction from Word Embedding Spaces (2021.findings-emnlp)
Copied to clipboard
Kelly Marchisio, Youngser Park, Ali Saad-Eldin, Anton Alyakin, Kevin Duh, Carey Priebe, Philipp Koehn
| Challenge: | Existing work in bilingual lexicon induction views word embeddings as vectors in Euclidean space. |
| Approach: | They propose to use word embeddings as nodes in a weighted graph to examine a node’s graph neighborhood without assuming a linear transform. |
| Outcome: | The proposed approaches are compared under different data conditions and show that they complement each other when combined. |
Estimating User Communication Styles for Spoken Dialogue Systems (2020.lrec-1)
Copied to clipboard
| Challenge: | a neural network estimation system for spoken dialogues can be used to estimate the communication style of a user's interaction, but this is rarely implemented in a live system. |
| Approach: | They propose a neural network approach to estimate the communication style of spoken interaction, namely elaborateness and directness. |
| Outcome: | The proposed method can estimate the elaborateness and directness of spoken interaction and improve the results with additional linguistic features. |
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Sentence embeddings are an important component of many natural language processing systems. |
| Approach: | They propose a self-supervised objective for learning universal sentence embeddings that does not require labelled training data. |
| Outcome: | The proposed approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. |
AMenDeD: Modelling Concepts by Aligning Mentions, Definitions and Decontextualised Embeddings (2024.lrec-main)
Copied to clipboard
| Challenge: | Contextualised Language Models (LMs) improve on word embeddings by encoding meaning of words in context. |
| Approach: | They propose to learn a unified embedding space in which all three types of representations can be integrated. |
| Outcome: | The proposed model outperforms existing approaches in ontology completion tasks. |
Do Word Embeddings Capture Spelling Variation? (2020.coling-main)
Copied to clipboard
| Challenge: | Using word embeddings, we analyze spelling variation in word embeds trained on Twitter and Reddit data. |
| Approach: | They propose a new perspective on the analysis of word embeddings by focusing on spelling variation. |
| Outcome: | The proposed analysis shows that word embeddings encode spelling variation patterns of various types to some extent, even when trained using the skipgram model. |
Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to quantify gender bias in word embeddings are not robust and cannot identify common types of bias. |
| Approach: | They propose to quantify gender bias by using cosine similarity to a pair of gender words and using analogies. |
| Outcome: | The proposed methods are not robust and cannot identify common types of bias, while analogies are unsuitable indicators. |
How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure (2023.tacl-1)
Copied to clipboard
| Challenge: | Competent speakers of a language know how likely a word w is to appear in a specific context . |
| Approach: | They use transformer-based large language models to generalize a novel noun argument . they show a bias to generalise based on linear order, instead of a linear order . |
| Outcome: | The proposed models perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training. |
Automated essay scoring with string kernels and word embeddings (P18-2)
Copied to clipboard
| Challenge: | Existing approaches to automatic essay scoring use low-level character n-gram features. |
| Approach: | They propose to combine string kernels and word embeddings for automatic essay scoring. |
| Outcome: | The proposed method outperforms state-of-the-art deep learning methods in Arabic dialect identification and native language identification tasks. |
Clustering Word Embeddings with Self-Organizing Maps. Application on LaRoSeDa - A Large Romanian Sentiment Data Set (2021.eacl-main)
Copied to clipboard
| Challenge: | Romanian is one of the understudied languages in computational linguistics, with few resources available for the development of natural language processing tools. |
| Approach: | They introduce a Large Romanian Sentiment Data Set which is composed of 15,000 positive and negative reviews collected from the largest Romanian e-commerce platform. |
| Outcome: | The proposed data set is composed of 15,000 positive and negative reviews from the largest Romanian e-commerce platform. |
Learning and Evaluating Character Representations in Novels (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in word embeddings have proven successful in learning entity representations from short texts but do not capture full book-level information. |
| Approach: | They propose two novel ways to learn fixed-length vector representations of characters from novels . they use graph neural network-based embeddings from a full corpus-based character network . |
| Outcome: | The proposed methods outperform text-based embeddings in four tasks. |
Knowing the Author by the Company His Words Keep (L18-1)
Copied to clipboard
| Challenge: | In traditional linguistics, there exists a famous saying that one should know a word by the company it keeps. |
| Approach: | They propose a method which uses word embeddings to identify pairwise relational features in the context of authorship attribution. |
| Outcome: | The proposed method is based on three literary corpora and shows that word similarity is a key feature in the authorship attribution task. |
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (N18-2)
Copied to clipboard
| Challenge: | Pre-trained word embeddings have proven to be invaluable for improving performance in natural language analysis tasks where large-scale parallel corpora cannot be obtained. |
| Approach: | They perform five sets of experiments to analyze when pre-trained word embeddings can be useful in NMT tasks. |
| Outcome: | The embeddings provide gains of up to 20 BLEU points in the most favorable setting. |
Encoding Sentiment Information into Word Vectors for Sentiment Analysis (C18-1)
Copied to clipboard
| Challenge: | Existing methods for embedding sentiment knowledge into word vectors are generally trained independently of the downstream task. |
| Approach: | They propose to encode sentiment knowledge into pre-trained word vectors to improve sentiment analysis. |
| Outcome: | The proposed method improves sentiment analysis on four popular sentiment datasets compared to benchmark methods. |
Subword Attention and Post-Processing for Rare and Unknown Contextualized Embeddings (2024.findings-naacl)
Copied to clipboard
| Challenge: | Word embeddings are useful, but struggle on rare and unknown words. |
| Approach: | They propose a rare/unknown embedding architecture that focuses on contextualized representations. |
| Outcome: | The proposed architecture improves performance in most intrinsic and downstream tasks. |
Learning Bias-reduced Word Embeddings Using Dictionary Definitions (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing word embeddings have undesirable gender, racial, and religious biases . DD-GloVe is a train-time debiasing algorithm that uses dictionary definitions based on word definitions. |
| Approach: | They propose a dictionary-guided loss function that encourages word embeddings to be similar to their relatively neutral dictionary definition representations. |
| Outcome: | The proposed algorithm can learn word embeddings by leveraging dictionary definitions. |
Enhancing General Sentiment Lexicons for Domain-Specific Use (C18-1)
Copied to clipboard
| Challenge: | Lexicon based methods for sentiment analysis rely on high quality polarity lexicons. |
| Approach: | They evaluate SentProp framework for inducing domain-specific polarities from word embeddings and use it to enhance a general-purpose lexicon for use in the political domain. |
| Outcome: | The proposed framework performs worse than the original lexicon in an out-domain task, showing that the words added and the polarity shifts applied are domain-specific and do not translate well to an out domain setting. |
Adversarial Contrastive Estimation (P18-1)
Copied to clipboard
| Challenge: | Noise contrastive estimation (NCE) is a general strategy used in word embeddings and translations for knowledge graphs. |
| Approach: | They propose to augment negative sampler into mixture distribution with adversarially learned sampler and to combine it with noise contrastive estimation (NCE) they observe faster convergence and improved results on multiple metrics. |
| Outcome: | The proposed model performs better on word embeddings, order embedds and knowledge graph embeddments and faster convergence and improved results on multiple metrics. |
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Unsupervised Anomaly Detection in Multi-Topic Short-Text Corpora (2023.eacl-main)
Copied to clipboard
| Challenge: | Unsupervised anomaly detection is a challenging task when the majority class is heterogeneous. |
| Approach: | They propose to use word embeddings to represent each sample by a dense vector and use a Mixture Model approach to detect which samples deviate the most from the underlying distributions of the corpus. |
| Outcome: | The proposed method is more efficient than state-of-the-art methods on real datasets. |
SOS: Systematic Offensive Stereotyping Bias in Word Embeddings (2022.coling-1)
Copied to clipboard
| Challenge: | Systematic Offensive Stereotyping (SOS) in word embeddings could lead to associating marginalised groups with hate speech and profanity. |
| Approach: | They propose a quantitative measure of the systematic offensive stereotyping (SOS) in word embeddings and validate it in most commonly used word embeds. |
| Outcome: | The proposed measure correlates with published statistics on online extremism, but does not explain hate speech detection models. |
Analyzing Continuous Semantic Shifts with Diachronic Word Similarity Matrices (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to analyze word sense proportions are insufficient for understanding semantic shifts . et al., 2018: semantic shift and its effects. |
| Approach: | They propose a framework for how semantic shifts occur over multiple time periods by using word embeddings. |
| Outcome: | The proposed framework can analyze semantic shifts over multiple time periods using word embeddings. |
A Retrofitting Model for Incorporating Semantic Relations into Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | Existing word embedding models mix semantic similarity with other types of relatedness. |
| Approach: | They propose a model that leverages relational knowledge available in a knowledge resource to improve word embeddings. |
| Outcome: | The proposed model improves word embeddings on synonymy, antonymy and hypernymy relations in WordNet and significantly improves lexical entailment detection task. |
Searching for the X-Factor: Exploring Corpus Subjectivity for Word Embeddings (P18-1)
Copied to clipboard
| Challenge: | Existing word embedding methods for natural language processing are limited in their ability to produce dense word embeds. |
| Approach: | They propose a word embedding SentiVec which is infused with sentiment information from a lexical resource and outperforms baselines on subjectivity-sensitive tasks. |
| Outcome: | The proposed word embedding SentiVec outperforms baselines on subjectivity-sensitive tasks. |
Incorporating Latent Meanings of Morphological Compositions to Enhance Word Embeddings (P18-1)
Copied to clipboard
| Challenge: | Existing word embedding methods learn semantic information at word level while neglecting meaningful inner structures of words like morphemes. |
| Approach: | They propose to use latent meanings of morphological compositions of words to train word embeddings. |
| Outcome: | The proposed models outperform baseline models on word similarity, syntactic analogy and text classification tasks. |
Multilingual Irony Detection with Dependency Syntax and Neural Models (2020.coling-main)
Copied to clipboard
Alessandra Teresa Cignarella, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, Paolo Rosso, Farah Benamara
| Challenge: | Several semantic and syntactic devices can be used to express irony, causing the incongruity, determine the clash and play the role of irony triggers within a text. |
| Approach: | They propose to exploit linguistic resources where syntax is annotated according to the Universal Dependencies scheme. |
| Outcome: | The proposed method exploits linguistic resources where syntax is annotated according to the Universal Dependencies scheme. |
On the Vector Representation of Utterances in Dialogue Context (L18-1)
Copied to clipboard
| Challenge: | In recent years, the representation of words as vectors in a vector space has gained a high degree of attention in the research community. |
| Approach: | They introduce a new language resource that represents dialogue utterances in vector space and captures the semantic meaning of those utterrances in the dialogue context. |
| Outcome: | The proposed model captures relevant semantic information by comparing them to manually annotated dialogue acts. |
Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media (N18-1)
Copied to clipboard
| Challenge: | Current approaches to Named Entity Recognition (NER) are effective in formal text, but they fail on informal text, where improper grammatical structures, spelling inconsistencies, and slang vocabulary prevail. |
| Approach: | They propose a multitask end-to-end bidirectional long short-term memory (BLSTM)-Conditional Random Field (CRF) network with two CRF classifiers and a feature extractor that transfers learning to a CRF for prediction. |
| Outcome: | The proposed models outperform the current state-of-the-art on the Workshop on Noisy User-generated Text 2017 dataset by 2.45% and 3.69%, establishing a more suitable approach for social media environments. |
Global Textual Relation Embedding for Relational Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing methods to learn textual relation embeddings are lacking in large open-domain corpora. |
| Approach: | They propose to learn a general-purpose embedding of textual relations using a large dataset from Freebase. |
| Outcome: | The proposed embedding can facilitate downstream tasks requiring relational understanding of the text. |
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word embeddings cannot model the dynamic nature of words’ semantics, i.e., the property of words to correspond to potentially different meanings. |
| Approach: | They propose a large-scale Word in Context dataset, called WiC, which is curated by experts and can be used to evaluate context-sensitive representations. |
| Outcome: | The proposed models outperform the standard evaluation dataset for the purpose and highlight their shortcomings. |
Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too! (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing topic models rely on probabilistic models to uncover themes within document collections, but are they the only option? |
| Approach: | They propose a way to cluster pre-trained word embeddings while incorporating document information for weighted clustering and reranking top words. |
| Outcome: | The proposed approach performs as well as classical topic models, but with lower runtime and computational complexity. |
Relation Induction in Word Embeddings Revisited (C18-1)
Copied to clipboard
| Challenge: | Existing approaches to relation induction are based on vector translations, but they are often inadequate for knowledge base completion. |
| Approach: | They propose to use Gaussian to explicitly model the variability of translations and Bayesian linear regression to encode the assumption that there is a linear relationship between the vector representations of related words. |
| Outcome: | The proposed models are based on translations but use Gaussian to model the variability of translations and encode soft constraints on the source and target words that may be chosen. |
Learning Word Meta-Embeddings by Autoencoding (C18-1)
Copied to clipboard
| Challenge: | Existing word embeddings have shown superior performance in numerous Natural Language Processing (NLP) tasks, however, their performances vary significantly across different tasks. |
| Approach: | They propose to combine distributed word embeddings to produce more accurate and complete meta-embeddings of words. |
| Outcome: | The proposed meta-embeddings outperform the state-of-the-art in multiple tasks. |
Field Embedding: A Unified Grain-Based Framework for Word Representation (2021.naacl-main)
Copied to clipboard
| Challenge: | Current methods focus on learning word embeddings while linguistic information is discarded after the learning. |
| Approach: | They propose a framework field embedding to jointly learn word and grain embedds by incorporating morphological, phonetic, and syntactical linguistic fields. |
| Outcome: | The proposed framework integrates morphological, phonetic, and syntactical linguistic fields to learn word embeddings and grain embedds. |
Improving Relation Extraction through Syntax-induced Pre-training with Dependency Masking (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies require modifications to existing baseline architectures to leverage syntactic information. |
| Approach: | They propose to leverage syntactic information to improve relation extraction by training a syntax-induced encoder on auto-parsed data through dependency masking. |
| Outcome: | The proposed approach outperforms baseline models and achieves state-of-the-art results on two English datasets. |
Encoders Help You Disambiguate Word Senses in Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Neural machine translation models can perform word sense disambiguation (WSD) however, it is unclear which component dominates the process of disambiguating words. |
| Approach: | They evaluate hidden states and investigate distributions of self-attention in NMT encoders and decoders to disambiguate word senses. |
| Outcome: | The proposed model outperforms encoder hidden states on large datasets . the model outpersforms decoders on large data sets . |
Autoencoding Improves Pre-trained Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work has shown that word embeddings are distributed in a narrow cone and that centering and projection can improve the accuracy of pre-trained word embeds without requiring additional training data. |
| Approach: | They propose to remove the top principal components from pre-trained word embeddings and center and project them onto principal component vectors to reinstate isotropy in the embeddable space. |
| Outcome: | The proposed method is equivalent to applying a linear autoencoder to minimize the squared L2 reconstruction error. |
Unequal Representations: Analyzing Intersectional Biases in Word Embeddings Using Representational Similarity Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Specifically, we probe contextualized and non-contextualized word embeddings for evidence of intersectional biases against Black women. |
| Approach: | They propose a representational similarity analysis approach to detect human-like biases in word embeddings using representational similarities analysis. |
| Outcome: | The proposed approach aligns with intersectionality theory, which states that multiple identity categories layer on top of each other to create unique modes of discrimination that are not shared by any individual category. |
comp-syn: Perceptually Grounded Word Embeddings with Color (2020.coling-main)
Copied to clipboard
Bhargav Srinivasa Desikan, Tasker Hull, Ethan Nadler, Douglas Guilbeault, Aabir Abubakar Kar, Mark Chu, Donald Ruggiero Lo Sardo
| Challenge: | Existing approaches to natural language processing ignore embodied sensory aspects of language. |
| Approach: | They propose a Python package that provides word embeddings based on Google Image search results. |
| Outcome: | The proposed package provides word embeddings based on the color distributions of Google Image search results. |
Constructing High Quality Sense-specific Corpus and Word Embedding via Unsupervised Elimination of Pseudo Multi-sense (L18-1)
Copied to clipboard
| Challenge: | Existing word embedding frameworks distinguish different senses of words by their contexts. |
| Approach: | They propose a framework for unsupervised corpus sense tagging which trains multi-sense word embeddings on a given corpus. |
| Outcome: | The proposed framework detects pseudo multi-senses without extra language resources without additional language resources. |
UnClE: Explicitly Leveraging Semantic Similarity to Reduce the Parameters of Word Embeddings (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to reduce word embedding parameters ignore semantic information . existing methods do not consider semantic information, allowing for performance degradation . |
| Approach: | They propose a method that leverages semantic similarity with weight sharing to reduce dimensionality of word embeddings. |
| Outcome: | The proposed method reduces word embedding parameters by more than 11x on a standard English-German dataset. |
Word Tour: One-dimensional Word Embeddings via the Traveling Salesman Problem (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing word embeddings are high-dimensional and consume considerable computational resources. |
| Approach: | They propose a method to decompose the desiderata of word embeddings into two parts, completeness and soundness, and focus on soundness. |
| Outcome: | The proposed method is extremely efficient and provides minimal means to handle word embeddings. |
Gender-preserving Debiasing for Pre-trained Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing methods for debiasing word embeddings have shown discriminative biases . word embeds learnt from social media have shown to encode racist, offensive and discriminative language usage. |
| Approach: | They propose a method that preserves gender-related information while removing stereotypical gender biases from pre-trained word embeddings. |
| Outcome: | The proposed method preserves gender-related information while removing stereotypical discriminative gender biases from pre-trained word embeddings. |
All-words Word Sense Disambiguation Using Concept Embeddings (L18-1)
Copied to clipboard
| Challenge: | Existing work on all-words word sense disambiguation (all-word WSD) uses word embeddings to identify the senses of words in documents. |
| Approach: | They propose a new concept embedding method to predict target word senses . concept embeds are constructed from concept tag sequences created from previous predictions . |
| Outcome: | The proposed concept embeddings improve Japanese all-words word sense disambiguation task. |
Habibi - a multi Dialect multi National Arabic Song Lyrics Corpus (2020.lrec-1)
Copied to clipboard
| Challenge: | Unlike western music, Arabic songs are poorly classified and the majority of the songs available online are classified under Modern Arabic Pop genre or what is now known as Franco-Arabic . |
| Approach: | They introduce Habibi the first Arabic Song Lyrics corpus for singers from 18 different Arabic countries. |
| Outcome: | The proposed corpus contains more than 30,000 Arabic song lyrics in 6 Arabic dialects for singers from 18 different arab countries. |
LSTMEmbed: Learning Word and Sense Representations from a Large Semantically Annotated Corpus with Long Short-Term Memories (P19-1)
Copied to clipboard
| Challenge: | Recent work has focused on vector representations which capture different meanings, i.e., senses, of words. |
| Approach: | They propose a bidirectional LSTM model which learns word senses from semantically annotated corpora by focusing on word order. |
| Outcome: | The proposed model achieves state-of-the-art on the SemEval-2014 word-to-sense similarity task and is available online at http://lcl.uniroma1.it/LSTMEmbed. |
Understanding Undesirable Word Embedding Associations (P19-1)
Copied to clipboard
| Challenge: | Word embeddings are often criticized for capturing undesirable word associations such as gender stereotypes. |
| Approach: | They propose to use subspace projection to debias vectors post hoc using a model that implicitly does matrix factorization to debunk gender bias. |
| Outcome: | The proposed test overestimates gender bias in word embeddings by using subspace projection, a method that is widely used in training. |
Retrofitting Word Representations for Unsupervised Sense Aware Word Similarities (L18-1)
Copied to clipboard
| Challenge: | Standard word embeddings lack the ability to distinguish senses of a word by projecting them to exactly one vector. |
| Approach: | They propose to retrofit standard word embeddings to produce sense-aware embeddable vectors using external resources as sense inventories. |
| Outcome: | The proposed method improves word similarity and relatedness scores on multiple word embeddings and established word similarities, sometimes up to an impressive margin of +0.15 Spearman correlation score. |
Measuring Intersectional Biases in Historical Documents (2023.findings-acl)
Copied to clipboard
Nadav Borenstein, Karolina Stanczak, Thea Rolskov, Natacha Klein Käfer, Natália da Silva Perez, Isabelle Augenstein
| Challenge: | digitised historical documents suffer from errors introduced by optical character recognition (OCR) and are written in an archaic language. |
| Approach: | They investigate the continuities and transformations of bias in Caribbean historical newspapers during the colonial era . they use distributional semantics models and word embeddings to measure gender, race, and intersectional biases. |
| Outcome: | The authors show that gender and racial biases are interdependent and their intersection triggers distinct effects. |
Streaming word similarity mining on the cheap (D18-1)
Copied to clipboard
| Challenge: | Existing methods to estimate word similarities are to embed words in vector space and then calculate similarities between corresponding vectors. |
| Approach: | They propose a method that explicitly counts second-order co-occurrences to estimate word similarities from streams. |
| Outcome: | The proposed method is scalable, converges rapidly, behaves robustly under parameter changes, and captures word similarities on par with state-of-the-art word embeddings. |
Dissecting Contextual Word Embeddings: Architecture and Representation (D18-1)
Copied to clipboard
| Challenge: | Existing work on learning contextual representations has used LSTM-based biLMs, but there is no reason to believe this is effective. |
| Approach: | They propose to use pre-trained bidirectional language models to learn contextual word embeddings for four NLP tasks and to use them to study the effects of architecture on endtask accuracy. |
| Outcome: | The proposed models outperform word embeddings for four NLP tasks and all learn representations that vary with network depth. |
Text Similarity Estimation Based on Word Embeddings and Matrix Norms for Targeted Marketing (N19-1)
Copied to clipboard
| Challenge: | Existing methods to estimate document similarity based on word embeddings are mediocre . a recent study compared word and sentence embedded documents to a similarity estimate using matrix norms. |
| Approach: | They propose to combine word embeddings with matrix norms to obtain a similarity estimate. |
| Outcome: | The proposed method produces superior results for most of the investigated matrix norms compared to the classical cosine measure and several other similarity estimates. |
Unsupervised Induction of Linguistic Categories with Records of Reading, Speaking, and Writing (N18-1)
Copied to clipboard
| Challenge: | a few researchers have shown that data traces from human processing can be used to improve NLP models. |
| Approach: | They propose to use data readily available for most languages to improve unsupervised induction . they find that english unsupervised POS induction achieves an error reduction of 1.5% . |
| Outcome: | The proposed model improves on Ontonotes domains with a word embeddings. |
Word Embedding-Based Automatic MT Evaluation Metric using Word Position Information (N19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics for machine translation are difficult to address word meaning because it is a surface-level metric. |
| Approach: | They propose to use word embeddings, sentence-level tf-idf, and cosine similarity between two word embeds as features, weight, and the distance between two features as features. |
| Outcome: | The proposed metric can evaluate machine translation based on word meaning . it achieves highest correlation with human judgment among several representative metrics. |
Aff2Vec: Affect–Enriched Distributional Word Representations (C18-1)
Copied to clipboard
| Challenge: | Affective word distributions are not well understood in literature. |
| Approach: | They propose a model that embeds affective word interpretations into enriched word embeddings. |
| Outcome: | The proposed model outperforms the state-of-the-art in word-similarity tasks and in emotion analysis, personality detection, and frustration prediction tasks. |
Towards preserving word order importance through Forced Invalidation (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models are insensitive to word order . performance on NLU tasks remains unchanged even after permuting the word . |
| Approach: | They propose a simple approach called Forced Invalidation to force the model to identify permuted sequences as invalid samples. |
| Outcome: | The proposed approach significantly improves the sensitivity of the models to word order on English NLU and QA tasks over BERT-based and attention-based models over word embeddings. |
On the Curious Case of l2 norm of Sense Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | l2 norm of sense embeddings encodes information related to frequency of that sense in the training corpus . l2-normal feature is useful for word-in-context (WiC) and word sense disambiguation (WSD) |
| Approach: | They propose to include the l2 norm of a sense embedding as a feature in a classifier to improve word sense learning methods that use static sense embeds. |
| Outcome: | The l2 norm of sense embeddings is a surprisingly effective feature for word sense related tasks such as word-in-context (WiC) and word sense disambiguation (WSD). |
Contextualized Embeddings for Enriching Linguistic Analyses on Politeness (2020.coling-main)
Copied to clipboard
| Challenge: | Current word embeddings in natural language processing do capture context and thus can be leveraged to enrich linguistic analyses. |
| Approach: | They propose a model which leverages pre-trained BERT to cluster contextualized representations of a word based on context in which it appears and labels of items it occurs in. |
| Outcome: | The proposed model can detect interpretable, finer-grained context patterns associated with (im)polite language. |
GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent pre-trained language models such as BERT have led to noticeable improvements in semantic similarity detection. |
| Approach: | They propose to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. |
| Outcome: | The proposed method improves on multiple semantic similarity datasets and shows that it is beneficial and currently missing from the original model. |
Rethinking Topic Modelling: From Document-Space to Term-Space (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a paper examines the use of document-based topic models for defining term interactions . term-based models outperform document-derived models by a large margin, the authors argue . |
| Approach: | They propose to reformulate topic distributions as latent factors in term similarity space . they use word embeddings built with wide context windows to test their approach . |
| Outcome: | The proposed model outperforms standard document-based models by a large margin . the proposed model is less sensitive to document formatting and more precise . |
Lacking the Embedding of a Word? Look it up into a Traditional Dictionary (2022.findings-acl)
Copied to clipboard
Elena Sofia Ruzzetti, Leonardo Ranaldi, Michele Mastromattei, Francesca Fallucchi, Noemi Scarpato, Fabio Massimo Zanzotto
| Challenge: | Word embeddings are powerful dictionaries, but they fail to give sense to rare words . a large body of research is devoted to devising ways to capture word meaning . |
| Approach: | They propose to use definitions retrieved from traditional dictionaries to build word embeddings for rare words. |
| Outcome: | The proposed methods outperform state-of-the-art methods for embeddings of unknown words . the proposed methods significantly outperformed the BERT method for OOV words compared to the proposed method . |
Why does PairDiff work? - A Mathematical Analysis of Bilinear Relational Compositional Operators for Analogy Detection (C18-1)
Copied to clipboard
| Challenge: | Existing methods for representing semantic relations between words are unclear . identifying relations between word and entity is important for NLP applications . |
| Approach: | They propose to compute the vector offset between word embeddings to represent relation between two words . they show that PairDiff is an uncorrelated bilinear operator that can be simplified to a linear form . |
| Outcome: | The proposed method is surprisingly accurate and can be used on multiple word embeddings. |
A Deterministic Algorithm for Bridging Anaphora Resolution (D18-1)
Copied to clipboard
| Challenge: | Existing methods for bridging anaphora resolution only consider NPs’ head nouns and thus do not capture the semantics of NP. |
| Approach: | They propose a deterministic approach to bridging anaphora resolution which represents the semantics of an NP based on its head noun and modifications. |
| Outcome: | The proposed approach achieves competitive results compared to the best system in Hou et al. (2013) which explores Markov Logic Networks to model the problem. |
Using Word Embeddings for Unsupervised Acronym Disambiguation (C18-1)
Copied to clipboard
| Challenge: | Scientific papers contain many acronyms and abbreviations. |
| Approach: | They propose a method to choose the contextual correct definition of an acronym . they learn word embeddings for all words in the corpus and compare them with weighted averages . |
| Outcome: | The proposed method outperforms (classical) cosine similarity in a set of scientific papers. |
SeVeN: Augmenting Word Embeddings with Unsupervised Relation Vectors (C18-1)
Copied to clipboard
| Challenge: | Word embeddings use fixed-dimensional vectors to represent the meaning of words. |
| Approach: | They propose a pipeline for learning relation vectors based on word vector averaging and an ad hoc autoencoder. |
| Outcome: | The proposed pipeline can capture aspects of word meaning complementary to word embeddings. |
Word Embedding Evaluation for Sinhala (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings are a key component of the natural language processing process. |
| Approach: | They evaluate three standard word embedding models for Sinhala language using two evaluation methods: intrinsic evaluation and extrinsic evaluation. |
| Outcome: | The proposed models performed best in the three evaluation tasks, while FastText and Glove showed the lowest accuracies. |
Exploring the Linear Subspace Hypothesis in Gender Bias Mitigation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for gender bias mitigation for word embeddings are based on pre-trained word embeds . however, the assumption that the bias subspace is linear is untested . |
| Approach: | They propose a method to isolate gender bias in word embeddings using pre-trained word embeds. |
| Outcome: | The proposed method eliminates gender bias in word embeddings but assumes bias subspace is linear . the proposed method has some drawbacks, but it is a good one for a non-linear analysis. |
A Comprehensive Comparison of Word Embeddings in Event & Entity Coreference Resolution. (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Coreference Resolution rely on word embeddings for word representation, but performance of different embeddables is largely overlooked. |
| Approach: | They frame their study in the context of Event and Entity Coreference Resolution (EvCR & EnCR) they examine whether there is a trade-off between performance and embedding size . |
| Outcome: | The embeddings achieve 86% of the performance of the largest model while being 1.2% of its size. |
Neural Activation Semantic Models: Computational lexical semantic models of localized neural activations (C18-1)
Copied to clipboard
| Challenge: | Neural activation models have been proposed to map word semantics to localized neural activations. |
| Approach: | They propose a computational model that estimates semantic similarity in the neural activation space and investigate its performance for various natural language processing tasks. |
| Outcome: | The proposed model performs better than state-of-the-art word embeddings for the task of semantic similarity estimation between very similar or very dissimilar words while performing well on other tasks such as entailment and word categorization. |
Imputing Out-of-Vocabulary Embeddings with LOVE Makes LanguageModels Robust with Little Cost (2022.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art NLP systems are brittle when faced with Out-ofVocabulary words . we present a framework that extends word embeddings and makes them robust to OOV . |
| Approach: | They propose a framework that extends existing word embeddings and makes them robust to OOV. |
| Outcome: | The proposed model performs better on original datasets and corrupted variants than previous competitors. |
Neural Networks approaches focused on French Spoken Language Understanding: application to the MEDIA Evaluation Task (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies have focused on English language and tasks, but few have explored the complexity of a SLU task. |
| Approach: | They propose to explore Neural Networks approaches for a French Spoken Language Understanding task. |
| Outcome: | The proposed approach outperforms classical Neural Network Architectures and achieves state-of-the-art results. |
Locality Preserving Sentence Encoding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embeddings can be used to learn sentence embedds on the sentence level. |
| Approach: | They propose a sentence embedding method that uses the inner product to compute semantic similarity between sentences. |
| Outcome: | The proposed method encodes sentences better in the sense of semantic structures. |
Inspecting the concept knowledge graph encoded by modern language models (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are used to solve tasks such as summarization and information retrieval. |
| Approach: | They propose to use word embeddings, text generators, context encoders to extract underlying knowledge graphs of nine influential language models. |
| Outcome: | The proposed model is able to encode word embeddings, text generators, and context encoders, but suffers from several inaccuracies. |
HG2Vec: Improved Word Embeddings from Dictionary and Thesaurus Based Heterogeneous Graph (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models that learn word embeddings rely on a large corpus of data . however, these models require massive time and space for data pre-processing and training . |
| Approach: | They propose a model that learns word embeddings utilizing only dictionaries and thesauri . they exploit a new context-focused loss model that models transitive relationships between word pairs . |
| Outcome: | The proposed model reaches the state-of-art on multiple word similarity and relatedness benchmarks. |
Improving Cross-Domain Chinese Word Segmentation with Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to Chinese word segmentation (CWS) are character-based and word-based . character-driven approaches use conditional random field models to label sequences, with complex hand-crafted discrete features. |
| Approach: | They propose a semi-supervised word-based approach to improve cross-domain Chinese word segmentation given a baseline segmenter. |
| Outcome: | The proposed model outperforms state-of-the-art approaches on five datasets covering domains in novels, medicine, and patent. |
C-Journal: A Journaling Application for Detecting and Classifying Cognitive Distortions Using Deep-Learning Based on a Crowd-sourced Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Cognitive distortions are negatively biased thinking patterns and erroneous self-statements resulting from and leading to logical errors in one’s internal reasoning. |
| Approach: | They propose to provide an open-source labeled dataset of 14 cognitive distortions consisting of 34370 entries collected via crowd-sourcing, user questionnaires, and re-purposing emotions dataset from social media. |
| Outcome: | The proposed model was built into a free journaling and mood-tracking mobile application that pinpoints cognitive distortions to the users. |
Simple Algorithms For Sentiment Analysis On Sentiment Rich, Data Poor Domains. (C18-1)
Copied to clipboard
| Challenge: | Standard word embedding algorithms learn vector representations from large corpora of text documents in unsupervised fashion. |
| Approach: | They propose an algorithm that learns word embeddings jointly with a classifier . their algorithm leverages document label information to learn vector representations of words . |
| Outcome: | The proposed algorithm has superior performance on domains with limited data compared to other methods. |
An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained word embeddings have been shown to improve the performance of neural networks across a wide variety of tasks. |
| Approach: | They propose two new metrics to understand the downstream reliability of word embeddings. |
| Outcome: | The proposed model can improve performance with slight changes to the training data, but it can also fail with multiple neural network architectures. |
Reducing Gender Bias in Abusive Language Detection (D18-1)
Copied to clipboard
| Challenge: | Abusive language detection models tend to be biased toward identity words of a certain group of people . recent studies have raised concerns about the robustness of such systems . |
| Approach: | They propose to use debiased word embeddings, gender swap data augmentation to reduce model bias . they also propose to fine-tune models with a larger corpus to correct such bias if needed . |
| Outcome: | The proposed methods reduce model bias by 90-98% and can be extended to correct model bias in other scenarios. |
Evaluation of Greek Word Embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings are the most popular input for many NLP tasks. |
| Approach: | They propose to use Greek word embeddings as an unsupervised learning tool . they use a Greek word analogy test set and a morphological test collection to evaluate word similarities . |
| Outcome: | The proposed model is able to create meaningful representations of Greek words . the proposed model can be adapted to Greek language and polysemy . |
Concept Space Alignment in Multilingual LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large language models generalize somewhat across languages, but it is unclear whether this is a result of improved, implicit alignment, or of something else, e.g., linguistic overlap or semi-parallel subsets of training data. |
| Approach: | They hypothesize that implicit alignment is the reason for generalization in multilingual large language models. |
| Outcome: | The proposed model generalizes well across languages, but lacks linearity. |
Relational Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to learn word embeddings rely on external knowledge bases . however, they are limited by the amount of available relational knowledge . |
| Approach: | They propose to encode relational knowledge in a separate word embedding . this is complementary to a standard word embedded from co-occurrence statistics . |
| Outcome: | The proposed word embedding is complementary to a standard word embed. |
Delta Embedding Learning (P19-1)
Copied to clipboard
| Challenge: | Unsupervised word embeddings have limitations to the semantics of words and inadequate fine-tuning of embedded word can lead to suboptimal performance. |
| Approach: | They propose a method that optimizes word embeddings by regularizing them incrementally to ensure they are tuned in an incremental way. |
| Outcome: | The proposed method improves performance on various NLP tasks and shows that it absorbs semantic information without "forging" |
Putting Words in Context: LSTM Language Models and Lexical Ambiguity (P19-1)
Copied to clipboard
| Challenge: | In language, a word can contribute a very different meaning depending on the context . lexical ambiguity involves both morphosyntactic and semantic aspects . |
| Approach: | They propose a method to probe hidden representations for lexical and contextual information about words. |
| Outcome: | The proposed method shows that both types of information are represented to a large extent, but there is room for improvement for contextual information. |
Misspelling Oblivious Word Embeddings (N19-1)
Copied to clipboard
Aleksandra Piktus, Necati Bora Edizel, Piotr Bojanowski, Edouard Grave, Rui Ferreira, Fabrizio Silvestri
| Challenge: | Existing word embeddings have limited applicability to malformed texts . misspellings are frequent and embeddable for words that have not been observed at training time . |
| Approach: | They propose a method to learn word embeddings that are resilient to misspellings . they use FastText with subwords to train embeddables on a new dataset . |
| Outcome: | The proposed method is tested on a publicly available dataset. |
Multi-source Neural Topic Modeling in Multi-view Embedding Spaces (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work has used pre-trained word embeddings to address data sparsity in short-text or small document collections. |
| Approach: | They propose a neural topic modeling framework using multi-view embedding spaces to improve topic quality and deal with polysemy. |
| Outcome: | The proposed framework improves topic quality and deal with polysemy. |
Debiasing with Sufficient Projection: A General Theoretical Framework for Vector Representations (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-trained vector representations can inadvertently encode undesirable social biases. |
| Approach: | They propose a framework for reducing bias by transforming vector representations to an unbiased subspace using sufficient projection. |
| Outcome: | The proposed framework mitigates bias across debiasing and fairness tasks and across various vector representation types, including word embeddings and output representations of transformer models. |
Massive vs. Curated Embeddings for Low-Resourced Languages: the Case of Yorùbá and Twi (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study shows that word embeddings can be useful for training downstream natural language processing tasks. |
| Approach: | They compare word embeddings obtained by word embeds from curated corpora with a language-dependent processing. |
| Outcome: | The proposed model compares word embeddings with word embeds from curated corpora and a language-dependent processing on two African languages. |
Delta-training: Simple Semi-Supervised Text Classification using Pretrained Word Embeddings (D19-1)
Copied to clipboard
| Challenge: | Pretrained word embeddings outperforms classifiers with randomly initialized word embeds, a new method is proposed for semi-supervised text classification. |
| Approach: | They propose a method that uses pretrained word embeddings to predict text classification . they use unlabeled data to build a classifier, and use early-stopping to improve performance . |
| Outcome: | The proposed method outperforms self-training and co-training frameworks on unlabeled data. |
Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space (D19-1)
Copied to clipboard
| Challenge: | Word embeddings have a notable property that word relationships can exist as linear substructures in the embeddable space. |
| Approach: | They propose an alternative way in which downstream models might learn these relationships: orthogonal and linear transformations. |
| Outcome: | The proposed model can learn such relationships as geometric translations, but there is no evidence that it is exclusively accurate. |
Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition (D19-1)
Copied to clipboard
| Challenge: | Existing work on name-switching focuses on word-level aspects but neglects subword-level characteristics shared across languages. |
| Approach: | They propose hierarchical meta-Embeddings that combine word-level and subword-level embeddings to create language-agnostic lexical representations. |
| Outcome: | The proposed model achieves state-of-the-art in English-Spanish code-switching scenarios. |
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference (N19-1)
Copied to clipboard
| Challenge: | Existing inference models that rely heavily on unsupervised single-word embeddings struggle to learn implied relationships between pairs of words. |
| Approach: | They propose to use word embeddings to learn and use background knowledge about implied relationships between words that are crucial for cross-sentence inference problems. |
| Outcome: | The proposed models gain 2.7% on the recently released SQuAD 2.0 and 1.3% on MultiNLI, and 8.8% on the adversarial SQu AD datasets. |
Adaptive Compression of Word Embeddings (2020.acl-main)
Copied to clipboard
| Challenge: | Distributed representations of words have been an indispensable component for natural language processing (NLP) tasks. |
| Approach: | They propose a method that uses a code-book approach to represent words as discrete codes such as (8, 5, 2, 4). |
| Outcome: | The proposed method makes the highly compressed word embeddings without hurting the task accuracy. |
Building Semantic Grams of Human Knowledge (2020.lrec-1)
Copied to clipboard
| Challenge: | Word senses are typically defined with textual definitions and put in context via lexical-semantic relations such as synonymy, antonymy, hypernymy, etc. |
| Approach: | They propose a slot-filler structure to define the meaning of words in terms of their prototypical semantic information. |
| Outcome: | The proposed model improves on a semantic similarity task and shows significant improvements over state-of-the-art embeddings. |
Single Training Dimension Selection for Word Embedding with PCA (D19-1)
Copied to clipboard
| Challenge: | Existing methods for word embeddings are limited by the definition of 'low' dimensionality, which is often used to train word embeds into low dimensional continuous vector space. |
| Approach: | They propose a method to select the number of dimensions for word embeddings using PCA. |
| Outcome: | The proposed method trains one embedding with a generous upper bound (e.g. 1,000) of dimensions and then removes the lesser dimensions one at a time while recording the embeddables’ performance on language tasks. |
A Co-Attentive Cross-Lingual Neural Model for Dialogue Breakdown Detection (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for dialogue breakdown detection do not focus on preventing dialogue breakdowns. |
| Approach: | They propose a model that integrates a pretrained cross-lingual language model and a co-attention network for dialogue breakdown detection. |
| Outcome: | The proposed model outperforms all previous approaches on evaluation metrics in Japanese and English tracks in Dialogue Breakdown Detection Challenge 4 . |
The Undesirable Dependence on Frequency of Gender Bias Metrics Based on Word Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have found word embeddings can capture semantic similarity but may be affected by word frequency. |
| Approach: | They find that word embeddings can capture semantic similarity but may be affected by word frequency . they compare this effect with an alternative metric based on Pointwise Mutual Information . |
| Outcome: | The proposed method does not depend on word frequency, but it does return female bias in low frequency words. |
He said “who’s gonna take care of your children when you are at ACL?”: Reported Sexist Acts are Not Sexist (2020.acl-main)
Copied to clipboard
Patricia Chiril, Véronique Moriceau, Farah Benamara, Alda Mari, Gloria Origgi, Marlène Coulomb-Gully
| Challenge: | Sexism is prejudice or discrimination based on a person's gender. |
| Approach: | They propose to use a French dataset annotated for sexism detection to characterize sexist content and to train deep learning experiments on tweets. |
| Outcome: | The proposed dataset is the first to be used for sexism detection in France and constitutes a first step towards offensive content moderation. |
Limbic: Author-Based Sentiment Aspect Modeling Regularized with Word Embeddings and Discourse Relations (D18-1)
Copied to clipboard
| Challenge: | Existing models for finding aspects and sentiments in opinionated texts ignore sentiments and are not supervised. |
| Approach: | They propose a probabilistic model that finds aspects and sentiments in opinionated texts . they use authors, discourse relations, and word embeddings to capture regularities . |
| Outcome: | The proposed model outperforms state-of-the-art models in topic cohesion and sentiment classification. |
Development of a Japanese Personality Dictionary based on Psychological Methods (2020.lrec-1)
Copied to clipboard
| Challenge: | a new approach to constructing a personality dictionary with psychological evidence is needed . we use abstract terms such as "sociable person" or "kind" to describe ourselves or others . |
| Approach: | They propose a Japanese personality dictionary with weights for Big Five traits . they collect personality words and use word embeddings to construct the dictionary . |
| Outcome: | The proposed approach is the first to have psychological evidence tolerant to NLP standards. |
Finely Tuned, 2 Billion Token Based Word Embeddings for Portuguese (L18-1)
Copied to clipboard
| Challenge: | A distributional semantics model is instrumental to improve the performance of many applications and processing tasks for any language. |
| Approach: | They propose to develop an advanced distributional model for Portuguese with the largest vocabulary and best evaluation scores published so far. |
| Outcome: | The proposed model has the largest vocabulary and the best evaluation scores published so far. |
Towards a Semi-Automatic Detection of Reflexive and Reciprocal Constructions and Their Representation in a Valency Lexicon (2020.lrec-1)
Copied to clipboard
| Challenge: | valency lexicons describe valencies of verbs in non-reflexive and non-reciprocal constructions . reflexive and reciprocal constructions are common morphosyntactic forms of verb . |
| Approach: | They propose a semi-automatic procedure to detect verbs with reflexive and reciprocal constructions in corpus data. |
| Outcome: | The proposed procedure detects verbs that form reflexive and reciprocal constructions in corpus data. |
BERT for Monolingual and Cross-Lingual Reverse Dictionary (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to find the proper word for a word are based on the surface form of words, but they cannot extract the semantic meaning. |
| Approach: | They propose a method to make BERT generate the target word for this task . cross-lingual reverse dictionary is the task to find the proper target word . |
| Outcome: | The proposed method can generate the target word for cross-lingual reverse dictionary task even without the parallel corpus. |
CARER: Contextualized Affect Representations for Emotion Recognition (D18-1)
Copied to clipboard
| Challenge: | Existing methods to model emotion-relevant content are based on rule-based and statistics-based approaches. |
| Approach: | They propose a semi-supervised graph-based algorithm to produce rich structural descriptors . they use word embeddings to evaluate the algorithm on emotion recognition tasks . |
| Outcome: | The proposed method outperforms state-of-the-art methods on emotion recognition tasks. |
HOTTER: Hierarchical Optimal Topic Transport with Explanatory Context Representations (2021.findings-emnlp)
Copied to clipboard
Sabine Wehnert, Christian Scheel, Simona Szakács-Behling, Maret Nieländer, Patrick Mielke, Ernesto William De Luca
| Challenge: | Natural language processing (NLP) is often the backbone of today’s systems for user interactions, information retrieval and others. |
| Approach: | They propose an extension to a specific emerging hybrid document distance metric which combines topic models and word embeddings. |
| Outcome: | The proposed method is competitive on public datasets and the language model BERT is used for a document categorization task. |
Hyperbolic Relevance Matching for Neural Keyphrase Extraction (2022.naacl-main)
Copied to clipboard
| Challenge: | Keyphrase extraction is a fundamental task in natural language processing that aims to extract a set of phrases with important information from a source document. |
| Approach: | They propose a hyperbolic matching model to explore keyphrase extraction in hyperbolical space using word embeddings from RoBERTa to capture hierarchical syntactic and semantic structures. |
| Outcome: | The proposed model outperforms the state-of-the-art models on six benchmark datasets and outperformed previous models. |
CLFD: A Novel Vectorization Technique and Its Application in Fake News Detection (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing work on fake news detection is limited due to the complex nature of the news . |
| Approach: | They propose a statistical approach for the generation of feature vectors to describe a document . they use class label frequency distance to boost machine learning methods . |
| Outcome: | The proposed method outperforms deep learning methods in large datasets while outperforming traditional methods. |
SimplifyUR: Unsupervised Lexical Text Simplification for Urdu (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for text simplification for Urdu rely on manual lexicons and simplified corpora, but are not applicable to the language. |
| Approach: | They propose an unsupervised method for automatic text simplification for Urdu using word embeddings and morphological features. |
| Outcome: | The proposed method achieves BLEU score of 80.15 and SARI score of 42.02 on simple text generated on simplified corpora and human evaluations for correctness, grammaticality, meaning-preservation and simplicity. |
Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings (2020.acl-main)
Copied to clipboard
| Challenge: | Contextualized representations have become the default for downstream NLP applications. |
| Approach: | They propose a method for converting from contextualized representations to static lookup-table embeddings and apply it to 5 popular pretrained models and 9 sets of pretrained weights. |
| Outcome: | The proposed methods show that pooling over many contexts significantly improves representational quality under intrinsic evaluation. |
Dirichlet-Smoothed Word Embeddings for Low-Resource Settings (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing count-based word embeddings are superseded by machine-learning methods like word2vec and GloVe, but in many settings there is not much text data available. |
| Approach: | They propose to use positive pointwise mutual information (PPMI) weighted co-occurrence matrices to compute word embeddings from a corpus using large amounts of text data. |
| Outcome: | The proposed method outperforms word2vec and the state-of-the-art for low-resource settings and obtains competitive results for Maltese and Luxembourgish. |
IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (2020.findings-emnlp)
Copied to clipboard
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, Pratyush Kumar
| Challenge: | In this paper, we present NLP resources for 11 major Indian languages . distributional representations are the cornerstone of modern NLP, authors say . |
| Approach: | They introduce NLP resources for 11 major Indian languages from two major language families . monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . they also compile a benchmark for Indian language NLU to evaluate their results . |
| Outcome: | The monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . the pre-trained language models are based on the compact ALBERT model . |
Grapheme-level Awareness in Word Embeddings for Morphologically Rich Languages (L18-1)
Copied to clipboard
| Challenge: | a study of inflectional and non-alphabetic languages shows word vectors are sparse in data sparsity due to the morphological system of a language and its syllables. |
| Approach: | They propose a grapheme-level coding procedure for neural word embedding that uses syllable characters to represent word-internal features. |
| Outcome: | The proposed model is more capable of representing functional and semantic similarities than syllable-level and word-level models. |
Language-Independent Tokenisation Rivals Language-Specific Tokenisation for Word Similarity Prediction (2020.lrec-1)
Copied to clipboard
| Challenge: | Language-independent tokenisation (LIT) methods that do not require labelled language resources or lexicons have gained popularity because of their compactness and ability to handle unseen or rare words. |
| Approach: | They empirically compare language-independent tokenisation methods with language-specific tokenisation (LST) methods using carefully created lexicons and training resources. |
| Outcome: | The proposed methods outperform LIT and LST on evaluation tasks across eight languages. |
R-AT: Regularized Adversarial Training for Natural Language Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, adversarial training is a popular and powerful regularization method in the natural language domain. |
| Approach: | They propose to regularize adversarial training via dropout by perturbing word embeddings . they find that R-AT can improve many models by reducing adversariality . |
| Outcome: | The proposed method can reduce the inconsistency between training and testing of models with dropout. |
Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained language models provide the foundations for state-of-the-art performance across a wide range of natural language processing tasks, including text classification. |
| Approach: | They compare the performance of a linear classifier based on word embeddings with a pre-trained language model, i.e., BERT, across a wide range of datasets and classification tasks. |
| Outcome: | The proposed method outperforms baselines in standard datasets with large training sets, but in settings with small training datasets it performs better. |
HECTOR: A Hybrid TExt SimplifiCation TOol for Raw Texts in French (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing systems for automatic text simplification (ATS) focus on lexical and syntactic transformations, but there is no end-to-end system for French. |
| Approach: | They propose to use word embeddings for lexical simplification and rule-based strategies for syntax and discourse adaptations to improve the complexity of texts. |
| Outcome: | The proposed system performs at lexical, syntactic and discourse levels according to automatic and humanevaluations. |
Automating Idea Unit Segmentation and Alignment for Assessing Reading Comprehension via Summary Protocol Analysis (2022.lrec-1)
Copied to clipboard
| Challenge: | In second language learning, summaries are among the most popular type of student assignments. |
| Approach: | They propose to revise the annotation guidelines to allow machine implementation of the new annotation guidelines. |
| Outcome: | The proposed algorithm achieves 0.789 precision and 0.844 recall over the L2WS 2021 corpus. |
Cross-Lingual Word Embeddings for Turkic Languages (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing techniques to align monolingual embeddings are difficult to use because of low resources. |
| Approach: | They propose to use existing techniques to align monolingual embedding spaces for Turkic, Uzbek, Azeri, Kazakh and Kyrgyz languages. |
| Outcome: | The proposed techniques outperform existing techniques on bilingual dictionaries and an extrinsic task. |
Multilingual Culture-Independent Word Analogy Datasets (2020.lrec-1)
Copied to clipboard
| Challenge: | In text processing, deep neural networks use word embeddings as an input. |
| Approach: | They propose to use benchmark datasets to compare the quality of word embeddings in text processing . they use a word analogy task in Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovenian, and Swedish . |
| Outcome: | The proposed datasets are culturally independent and cross-lingual for the languages used. |
Multiplex Word Embeddings for Selectional Preference Acquisition (D19-1)
Copied to clipboard
| Challenge: | Existing word embeddings are limited in their ability to represent fixed vectors . instead, they incorporate relational dependencies of different words into their embeddables - a limitation that is addressed by a multiplex model . |
| Approach: | They propose a word embedding model which incorporates relational dependencies of different words into their embeddables. |
| Outcome: | The proposed model can be easily extended according to various relations among words. |
A Hmong Corpus with Elaborate Expression Annotations (2022.lrec-1)
Copied to clipboard
| Challenge: | SCH is the first substantial corpus to be annotated for elaborate expressions . a plurality of speakers are located in China, but many Hmong speakers left Laos as refugees . |
| Approach: | They describe the first publicly available corpus of Hmong, a minority language of China, Vietnam, Laos, Thailand, and various countries in Europe and the Americas. |
| Outcome: | The first publicly available corpus of Hmong is scraped from a long-running Usenet newsgroup . it is the first substantial corpus to be annotated for elaborate expressions . |
Neural-DINF: A Neural Network based Framework for Measuring Document Influence (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods to measure scholarly impact of documents without citations only consider word frequency change. |
| Approach: | They propose a neural network framework that measures document influence without citations by using word frequency changes and word semantic shifts. |
| Outcome: | The proposed model outperforms existing models on document influence evaluation without citations. |
All That Glitters is Not Gold: A Gold Standard of Adjective-Noun Collocations for German (2020.lrec-1)
Copied to clipboard
| Challenge: | Using the GerCo dataset, we identify adjective-noun collocations in German and compare them with statistical associations measures. |
| Approach: | They present a GerCo dataset of adjective-noun collocations for German, such as alter Freund ‘old friend’ and tiefe Liebe ‘deep love’. |
| Outcome: | The GerCo dataset contains 4,732 positive and negative instances of collocations and covers all 16 semantic classes of adjectives defined in the German wordnet GermaNet. |
Linguistically-Informed Self-Attention for Semantic Role Labeling (D18-1)
Copied to clipboard
| Challenge: | Existing models of semantic role labeling use no explicit linguistic features. prior work has shown that syntax trees can dramatically improve SRL decoding. |
| Approach: | They propose a neural network model that incorporates syntax using only raw tokens . they show that LISA out-performs the state-of-the-art with contextually-encoded word representations a 1.0 F1 on newswire and 2.0 F1 in out-of domain text . |
| Outcome: | The proposed model outperforms the state-of-the-art model with word embeddings and predicted predicates. |
Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings (2022.acl-long)
Copied to clipboard
| Challenge: | Contextualized embeddings are expensive and resource-demanding, hence environmentally unfriendly. |
| Approach: | They propose a method to convert contextualized embeddings from pre-trained models into static embeddables using synonym knowledge and weighted vector distribution. |
| Outcome: | The proposed method outperforms baseline embeddings by a large margin through extrinsic and intrinsic tasks. |
Annotating Educational Questions for Student Response Analysis (L18-1)
Copied to clipboard
| Challenge: | a new taxonomy and annotated educational corpus of questions is proposed for question answering systems. |
| Approach: | They propose a taxonomy and annotated educational corpus of questions that can be used in automatic questions classification systems. |
| Outcome: | The proposed approach achieves a weighted F1-score of 0.511, overtaking the baseline by 12%. |
Evaluating Performance of Pre-trained Word Embeddings on Assamese, a Low-resource Language (2024.lrec-main)
Copied to clipboard
| Challenge: | Word embeddings are not explored in high-resource languages such as Assamese, where resources are limited. |
| Approach: | They propose to use assamese pre-trained word embeddings for sequence labeling tasks such as Parts-of-speech and Named Entity Recognition to evaluate their performance. |
| Outcome: | The proposed embeddings outperform the existing methods on Parts-of-speech and Named Entity Recognition tasks. |
Word2Sense: Sparse Interpretable Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Word2Sense embeddings are interpretable, but they are sparse and fast to compute . a unitary rotation can be applied to many of these embeddables retaining their utility for computational tasks while changing the values of individual coordinates. |
| Approach: | They propose an unsupervised method to generate Word2Sense word embeddings that are interpretable. |
| Outcome: | The proposed method compares well with other unsupervised word embeddings on NLP tasks. |
Predicting Humorousness and Metaphor Novelty with Gaussian Process Preference Learning (P19-1)
Copied to clipboard
| Challenge: | Inability to quantify key aspects of creative language is a frequent obstacle to natural language understanding. |
| Approach: | They propose a Bayesian approach for predicting humorousness and metaphor novelty using Gaussian process preference learning (GPPL) . |
| Outcome: | The proposed approach achieves a Spearman’s of 0.56 against gold using word embeddings and linguistic features. |
Collocation Classification with Unsupervised Relation Vectors (P19-1)
Copied to clipboard
| Challenge: | Existing methods for relation classification are based on word embeddings, but they pose a greater challenge than syntactic and semantic relations. |
| Approach: | They propose a distributional landscape based on word embeddings as a suitable basis for relation classification of collocations . they also conduct experiments on a subset of this benchmark . |
| Outcome: | The proposed dataset is compared to the well known DiffVec dataset and shows that it is more efficient than the standard methods. |
Multi-lingual Mathematical Word Problem Generation using Long Short Term Memory Networks with Enhanced Input Features (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for multi-lingual MWP generation are incapable of identifying language specific constraints, especially in morphologically rich yet low resource languages such as Sinhala and Tamil. |
| Approach: | They propose to use a long-term memory network to generate elementary level MWPs by adding character embeddings, word embedds and Part of Speech (POS) tag embeddements to the network. |
| Outcome: | The proposed model generates elementary level MWPs while satisfying language specific constraints while providing attention for numerical values and units. |
Time-Aware Word Embeddings for Three Lebanese News Archives (2020.lrec-1)
Copied to clipboard
| Challenge: | a large corpus of newspaper archives has been generated, but historians have struggled to analyze it manually for decades. |
| Approach: | They propose to train word embeddings from three large Lebanese news archives, which collectively consist of 609,386 scanned newspaper images and span 151 years. |
| Outcome: | The embeddings are trained using a Google Tesseract 4.0 OCR engine and a benchmark of analogy tasks to evaluate their accuracy. |
Urban Dictionary Embeddings for Slang NLP Applications (2020.lrec-1)
Copied to clipboard
| Challenge: | a new set of word embeddings is released to improve word embedment performance . word embeds provide useful representations of meanings of words in vectors . |
| Approach: | They present a set of word embeddings trained on Urban Dictionary . they show they have high performance across a range of common word embeding evaluations . |
| Outcome: | The first set of word embeddings trained on Urban Dictionary has high performance . the embeddables perform better on a range of common word evaluation tasks . |
Expert Concept-Modeling Ground Truth Construction for Word Embeddings Evaluation in Concept-Focused Domains (2020.coling-main)
Copied to clipboard
| Challenge: | a novel method for concept-modeling ground truths is developed to evaluate the application of word embeddings in concept-focused textual domains. |
| Approach: | They propose a domain expert-controlled procedure for the construction of concept-modeling ground truths with the aim of evaluating the application of word embeddings. |
| Outcome: | The proposed method evaluates the application of word embeddings in concept-focused textual domains. |
On the Correlation of Word Embedding Evaluation Metrics (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings are geometrical representations of word paradigmatics and syntagmatics. |
| Approach: | They propose to investigate evaluation metrics on various datasets to find correlations . they propose a fast solution to select the best word embeddings among many others . |
| Outcome: | The proposed method could be used to select the best word embeddings among many others. |
Embedding Words in Non-Vector Space with Unsupervised Graph Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | GraphGlove is an unsupervised graph word representations that are learned end-to-end. |
| Approach: | They propose a method to learn weighted graph word representations end-to-end using a weighteable weighte . they adopt a hierarchical graph representation method and modify the GloVe training algorithm to learn graph representations. |
| Outcome: | The proposed method outperforms vector-based methods on word similarity and analogy tasks. |
Augmenting NLP models using Latent Feature Interpolations (2020.coling-main)
Copied to clipboard
| Challenge: | Existing data augmentation methods with a large number of parameters are prone to over-fitting and often fail to capture the underlying input distribution. |
| Approach: | They propose a data augmentation technique that uses embeddings and hidden layer representations to construct virtual examples. |
| Outcome: | The proposed method outperforms existing methods in terms of accuracy and robustness to weight pruning. |
Graph Based Semi-Supervised Learning Approach for Tamil POS tagging (L18-1)
Copied to clipboard
| Challenge: | Parts of Speech (POS) tagging is challenging for low resourced languages such as Tamil . low resource Tamil does not have large POS annotated corpus to build good quality POS taggers using supervised machine learning techniques. |
| Approach: | They propose a graph-based semi-supervised learning approach to classify unlabelled data using a small POS labelled data set. |
| Outcome: | The proposed method achieves 0.8743 over 0.7333 produced by a CRF tagger for the same limited size corpus. |
Offensive Language Detection Using Brown Clustering (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study shows that Brown clustering is of little use when distinguishing word polarity in sentiment analysis tasks. |
| Approach: | They investigate the use of Brown clustering for offensive language detection . they train Brown clusters separately on positive and negative sentiment data, then combine it into a single complex feature per word . |
| Outcome: | The proposed method improves offensive language detection when used as the only feature or with words or character n-grams. |
Dependency Parsing for Urdu: Resources, Conversions and Learning (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing treebanks for Urdu are under-resourced due to lack of resources. |
| Approach: | They propose to convert existing treebanks into a common format that is based on Universal Dependencies. |
| Outcome: | The proposed format outperforms the MaltParser and a transition-based BiLSTM parser with word embeddings and significantly improves parsing accuracy. |
FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on word embeddings and pre-trained language models has shown the large impact of language representations on natural language processing (NLP) models across tasks and domains. |
| Approach: | They propose feature-based adversarial meta-embeddings with an attention function that is guided by word-specific properties, such as shape and frequency, to handle subword-based embeddings. |
| Outcome: | The proposed model improves performance in downstream tasks even with word embeddings from transformers. |
GAProtoNet: A Multi-head Graph Attention-based Prototypical Network for Interpretable Text Classification (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models with black-box nature obscure decision-making process and lack interpretability. |
| Approach: | They propose a multi-head graph attention-based prototypical network that uses a vector and prototypes to learn an interpretable prototypical representation. |
| Outcome: | The proposed model achieves superior results without sacrificing the accuracy of the original black-box LMs. |
Interpretable Word Embeddings via Informative Priors (D19-1)
Copied to clipboard
| Challenge: | Existing word embeddings lack interpretability and are unsupervised . this limitation limits their use within computational social science and digital humanities. |
| Approach: | They propose to use informative priors to create interpretable dimensions for probabilistic word embeddings using a priori model. |
| Outcome: | The proposed models capture latent semantic concepts better than or on-par with the current state of the art while maintaining the simplicity and generalizability of priors. |
SenSALDO: Creating a Sentiment Lexicon for Swedish (L18-1)
Copied to clipboard
| Challenge: | sentiment analysis has seen an explosive expansion over the last decade or so . many theoretical and methodological questions remain unanswered and resource gaps unfilled . |
| Approach: | They develop a sentiment lexicon for written (standard) Swedish using an existing dataset . they assign a real value sentiment score in the range [-1,1] and produce a label for it . |
| Outcome: | The proposed sentiment lexicon is an open source resource from the Swedish Language Bank . it is based on an existing gold standard dataset and is available from Sprkbanken . |
Query Obfuscation by Semantic Decomposition (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods to protect privacy of search engine users by decomposing queries using semantically related and unrelated distractor terms are not available to most web search engines. |
| Approach: | They propose a method to protect the privacy of search engine users by decomposing queries using semantically related and unrelated distractor terms. |
| Outcome: | The proposed method can reconstruct search results relevant to the original query term without compromising the privacy of the search engine users. |
The Paradigm Discovery Problem (2020.acl-main)
Copied to clipboard
| Challenge: | a paradigm discovery problem is a task of learning an inflectional morphological system from unannotated sentences. |
| Approach: | They formalize the paradigm discovery problem and develop evaluation metrics for judging systems . they use word embeddings and string similarity to cluster forms by cell and by paradigm . |
| Outcome: | The proposed system suggests clustering by cell across different inflection classes is the most pressing challenge for future work. |
Are Word Embeddings Really a Bad Fit for the Estimation of Thematic Fit? (2020.lrec-1)
Copied to clipboard
| Challenge: | In recent years, vectors derived from neural network training have replaced count-based distributional semantic models as a de facto standard for word representation in NLP. |
| Approach: | They propose to evaluate count models and word embeddings on thematic fit estimation by taking into account a larger number of parameters and verb roles and introducing dependency-based embedders in the comparison. |
| Outcome: | The proposed model outperforms count models and word embeddings in thematic fit estimation tasks while introducing dependency-based embedders. |
BiLSTM-CRF for Persian Named-Entity Recognition ArmanPersoNERCorpus: the First Entity-Annotated Persian Dataset (L18-1)
Copied to clipboard
| Challenge: | Named-entity recognition (NER) is a natural language processing component that aims to identify all the "named entities" (NEs) in an unstructured text. |
| Approach: | They propose a deep learning approach for name-entity recognition in Persian . they publicize an entity-annotated Persian dataset and train word embeddings . |
| Outcome: | The proposed approach achieves a 77.45% CoNLL F 1 score for Persian NER based on a deep learning architecture and pre-trained word embeddings. |
A Structure-Aware Generative Adversarial Network for Bilingual Lexicon Induction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Bilingual lexicon induction (BLI) is the task of inducing word translations with a learned mapping function that aligns monolingual word embedding spaces in two different languages. |
| Approach: | They propose a model that explicitly captures multiple topological structure information to achieve accurate bilingual lexicon induction. |
| Outcome: | The proposed model captures multiple topological structure information to achieve accurate BLI on a public dataset. |
Named Entity Recognition Only from Word Embeddings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing named entity recognition systems require large amounts of human annotated training data. |
| Approach: | They propose a fully unsupervised named entity recognition model which takes clues from pre-trained word embeddings. |
| Outcome: | The proposed model can be trained on two CoNLL benchmark datasets without annotating lexicon or corpus. |
Estimating Mutual Information Between Dense Word Embeddings (2020.acl-main)
Copied to clipboard
| Challenge: | Word embedding-based similarity measures are among the top-performing methods on unsupervised semantic textual similarity tasks. |
| Approach: | They propose a word embedding similarity measure that is a viable alternative to correlations and gives an excellent signal that correlates well with human judgements of similarity. |
| Outcome: | The proposed model outperforms existing state-of-the-art methods on hard NLP tasks. |
Synthetic Data for English Lexical Normalization: How Close Can We Get to Manually Annotated Data? (2020.lrec-1)
Copied to clipboard
| Challenge: | Social media data is a valuable data resource for natural language processing tasks. |
| Approach: | They propose to adapt input text to a more standard form, a task also referred to as normalization. |
| Outcome: | The proposed system scores 94.29 accuracy on the test data compared to 95.22 when trained on human-annotated data. |
SAPPHIRE: Simple Aligner for Phrasal Paraphrase with Hierarchical Representation (2020.lrec-1)
Copied to clipboard
| Challenge: | Monolingual phrase alignment is a fundamental problem in natural language understanding and crucial technique in various applications. |
| Approach: | They propose a simple Aligner for Phrasal Paraphrase with HIerarchical REpresentation that uses word embeddings to train phrase alignments. |
| Outcome: | The proposed algorithm outperforms the previous methods and establishes the state-of-the-art. |
Adaptive Axes: A Pipeline for In-domain Social Stereotype Analysis (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to quantify social stereotypes have struggled to capture the variability in stereotypes across conceptual domains for the same social group. |
| Approach: | They propose to use text embedding models and adaptive semantic axes to recover stereotypes from contextual representations by using large language models. |
| Outcome: | The proposed pipeline surpasses token-based methods in capturing in-domain framing and tracks stereotypes along domain-specific semantic axes for in- domain texts. |
Projective Methods for Mitigating Gender Bias in Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Mitigating gender bias in NLP has a long history tied to debiasing static word embeddings. |
| Approach: | They propose a masked language modelling task where content is developed around known social stereotypes and a projective debiasing method is used to reduce bias. |
| Outcome: | The proposed methods reduce intrinsic bias and mitigat observed bias in a downstream setting, but the two outcomes are not necessarily correlated. |
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |
Towards Robust In-Context Learning for Machine Translation with Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate the effectiveness of our method, particularly in domain adaptation. |
| Approach: | They propose a method to retrieve translation pairs as demonstrations from an additional datastore to guide translation without updating the LLMs. |
| Outcome: | The proposed method reduces noise and improves translation performance in domain adaptation. |
3R: Enhancing Sentence Representation Learning via Redundant Representation Reduction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve sentence representations lack fine-grained guidance on reducing redundant information. |
| Approach: | They propose a method that dynamically identifies redundant information from a dimensional perspective and trains the SRL model to redistribute semantics on different dimensions. |
| Outcome: | The proposed method improves sentence representations on seven semantic text similarity benchmarks. |
Towards Author-informed NLP: Mind the Social Bias (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of text understanding fail when opinions are conveyed implicitly or sarcastically. |
| Approach: | They propose to model user contexts within a social embedding space that was learned from the Twitter network at large-scale. |
| Outcome: | The proposed model improves generalization of stance prediction and toxicity detection, and also toxicity and incivility detection. |
NormXLogit: The Head-on-Top Never Lies (2025.emnlp-main)
Copied to clipboard
| Challenge: | NormXLogit is a novel approach for assessing the significance of input tokens based on word embeddings . |
| Approach: | They propose a novel method for assessing the significance of input tokens based on the input and output representations associated with each token. |
| Outcome: | The proposed method outperforms gradient-based methods in faithfulness and offers competitive performance compared to leading architecture-specific techniques. |
Words that make SENSE: Sensorimotor Norms in Learned Lexical Token Representations (2026.findings-acl)
Copied to clipboard
| Challenge: | Empirical studies suggest that comprehending action, perceptual and abstract concepts elicits rapid, automatic activity in modality-specific brain areas. |
| Approach: | They propose a model that predicts Lancaster sensorimotor norms from word lexical embeddings. |
| Outcome: | The proposed model predicts Lancaster sensorimotor norms from word lexical embeddings. |